Systolic array-friendly data placement and control

ABSTRACT

The present disclosure relates to an accelerator for systolic array-friendly data placement. The accelerator may include: a systolic array comprising a plurality of operation units, wherein the systolic array is configured to receive staged input data and perform operations using the staged input to generate staged output data, the staged output data comprising a number of segments; a controller configured to execute one or more instructions to generate a pattern generation signal; a data mask generator; and a memory configured to store the staged output data using the generated masks. The data mask generator may include circuitry configured to: receive the pattern generation signal from the controller, and, based on the received signal, generate a mask corresponding to each segment of the staged output data.

BACKGROUND

Machine learning has been widely used in various areas. Convolutionalneural network (CNN) is one type of a neural network widely applied onmachine learning. CNNs are used in a variety of applications includingimage processing, speech recognition, game play, robotics, etc.Therefore, improving efficiency for processing CNNs is important toincrease overall neural network execution performance.

Systolic arrays are a popular architecture for matrix-matrixmultiplication. In order for a systolic array to perform matrix-matrixmultiplication, both input matrices have to be staged to a particularformat, which is usually done by line-buffers. However, line buffers arecostly, and three may be required (two for input, and one for output).The overhead resulting from the line-buffers is within the same order ofmagnitude of systolic array itself. Removing the line-buffers mayrequire the input data to be stored in memory with particular placementpattern. In certain applications, the intermediate output of last layerafter general matrix multiply (GeMM) may need to maintain the sameplacement format as the input data, which is costly in terms of controland latency. Thus, there is a trade-off between the hardware cost ofline-buffers (or equivalent module), or the cost of additional controland latency overhead.

SUMMARY OF THE DISCLOSURE

Embodiments of the present disclosure provide a system for systolicarray-friendly data placement. An accelerator may include: a systolicarray comprising a plurality of operation units, wherein the systolicarray is configured to receive staged input data and perform operationsusing the staged input to generate staged output data, the staged outputdata comprising a number of segments; a controller configured to executeone or more instructions to generate a pattern generation signal; a datamask generator; and a memory configured to store the staged output datausing the generated masks. The data mask generator may include circuitryconfigured to: receive the pattern generation signal from thecontroller, and, based on the received signal, generate a maskcorresponding to each segment of the staged output data.

Additional features and advantages of the disclosed embodiments will beset forth in part in the following description, and in part will beapparent from the description, or may be learned by practice of theembodiments. The features and advantages of the disclosed embodimentsmay be realized and attained by the elements and combinations set forthin the claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary convolutional neural network (CNN)operation.

FIG. 2A illustrates an exemplary neural network acceleratorarchitecture, consistent with embodiments of the present disclosure.

FIG. 2B illustrates an exemplary neural network accelerator corearchitecture, consistent with embodiments of the present disclosure.

FIG. 2C illustrates an exemplary operation unit, consistent withembodiments of the present disclosure.

FIG. 2D illustrates a schematic diagram of an exemplary cloud systemincorporating a neural network accelerator, consistent with embodimentsof the present disclosure.

FIG. 3 illustrates an exemplary systolic array, consistent withembodiments of the present disclosure.

FIG. 4 illustrates a flow diagram of an exemplary process for performingoperations using a systolic array, consistent with embodiments of thepresent disclosure.

FIG. 5A illustrates an example data saved in a typical format,consistent with embodiments of the present disclosure.

FIG. 5B illustrates an example of data saved in a staged format,consistent with embodiments of the present disclosure.

FIG. 6 illustrates the input and output in memory of a systolic arraysystem.

FIG. 7 illustrates an example a system implementing a systolic array,consistent with embodiments of the present disclosure.

FIG. 8 illustrates an exemplary stride of a data mask module, consistentwith embodiments of the present disclosure.

FIG. 9A illustrates another exemplary system for implementing a systolicarray, consistent with embodiments of the present disclosure.

FIG. 9B illustrates an exemplary data structure, consistent withembodiments of the present disclosure.

FIG. 10 illustrates exemplary memory transactions executed by a systemimplementing a systolic array, consistent with embodiments of thepresent disclosure.

FIG. 11 illustrates a flowchart of an exemplary method for implementinga systolic array, consistent with embodiments of the present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings. The followingdescription refers to the accompanying drawings in which the samenumbers in different drawings represent the same or similar elementsunless otherwise represented. The implementations set forth in thefollowing description of exemplary embodiments do not represent allimplementations consistent with the disclosed subject matter. Instead,they are merely examples of apparatuses and methods consistent withaspects related to the invention as recited in the appended claims.

Systolic array refers to a network of coupled processing units. Systolicarrays require data to be staged at input, e.g., such that each row ofthe systolic array receives data delayed by one clock cycle, and then tobe transformed back to their original state at output. This is oftendone using staging buffers. However, staging buffers are large (e.g.,N²/2), and multiple staging buffers may be required, e.g., two stagingbuffers for an input and one staging buffer for an output, which is1.5N² in total. These staging buffers occupy a silicon area of a similarscale to the systolic array itself.

The embodiments of the present disclosure provide systolic-aware dataplacement, for example, by using masked writes to store merged outputdata in memory. Accordingly, the disclosed embodiments overcome thepitfalls associated with the use of staging buffers. For example, thedisclosed embodiments may enable mask generation for systolic arraysduring a general matrix multiplication (GeMM) computations without astaging buffer.

FIGS. 1 and 2A-2D provide a brief overview of convolutional neuralnetworks and accelerator architectures.

FIG. 1 illustrates an exemplary convolutional neural network (CNN)operation. In this exemplary operation, input data 102 such asactivations are structured as a set of two-dimensional (2D) feature mapsacross a number (e.g., C) of channels. Each of the two-dimensionalfeature maps can be referred to as a channel. As shown in FIG. 1, inputdata 102 (e.g., an image) has C number of feature maps, and one channelof input data 102 has a size of H×W. Therefore, the size of the inputdata 102 can be H×W×C.

In FIG. 1, input data 102 can be convolved with a filter 104. Differentfeature maps of input data 102 may have different parameters such asweights, bias terms, etc. while one feature map can share the sameparameters. Therefore, each filter 104 can have C number of channelscorresponding to C number of feature maps of input data 102. Eachchannel of filter 104 can slide over a corresponding feature map ofinput data 102. As shown in FIG. 1, each channel of filter 104 has asize S×R, and the size of one filter 104 can be S×R×C. Here, a windowsliding over input data 102 for a convolution operation can have thesize S×R. In this example, input data 102 is convolved with K number offilters 104_1 to 104_k.

When a first channel of a first filter 104_1 slides on a first featuremap of input data 102 for a convolution operation, the first channel offirst filter 104_1 is multiplied to receptive fields such as b1 to b3 inthe first feature map of input data 102. Receptive fields b1 to b3 canbe defined to partially overlap with adjacent receptive fields. Forexample, first receptive field b1 partially overlaps with secondreceptive field b2 and third receptive field b3 as shown in FIG. 1.Receptive fields for the rest of feature maps of input data 102 can bedefined corresponding to receptive fields of the first feature map.Therefore, each of first receptive field b1 to third receptive field b3has C number of channels. When each feature map of input data 102 has Bnumber of receptive fields, it can be considered that, in the presentdisclosure, the input data 102 comprises B number of work items each ofwhich includes C number of channels for illustration purposes. Here, Cchannels of each work item can have a size corresponding to a size S×Rof the receptive field.

One output value can be generated by multiplying first filter 104_1 withfirst receptive field b1 of input data 102 and by summing multiplicationresults for C number of channels. For example, first output value r1_c1can be generated by multiplying each channel of first filter 104_1 andcorresponding feature map in the first receptive field b1 of input data102 and by summing multiplication results from C number of channels. Bymultiplying first filter 104_1 and second receptive field b2 of inputdata 102 channel by channel and by summing multiplication results for Cnumber of channels, second output value r2_c1 can be generated. B numberof output values including first output value r1_c1 and second outputvalue r1_c2 generated by sliding first filter 104_1 over input data 102can constitute a first channel of output data 106.

Similarly, B number of output values can be generated by convolvingsecond filter 104_2 on input data 102 and can constitute a secondchannel of output data 106. B number of output values can also begenerated by convolving K^(th) filter 104_k on input data 102 and canconstitute a K^(th) channel of output data 106. For example, firstoutput value r1_ck on the K^(th) channel can be generated by multiplyingeach channel of K^(th) filter 104_1 and corresponding feature map in thefirst receptive field b1 of input data 102 and by summing multiplicationresults for C number of channels. As shown in FIG. 1, output data 106can have K number of channels corresponding to the number of filters 104and each channel has a size of H′×W′. Therefore, the size of the outputdata 106 can be H′×W′×K. In this example, H′×W′ can be equal to thenumber of work items of input data 102, i.e., B.

In some embodiments, output data 106 can be intermediate output data ofa convolutional operation. In some embodiments, output data 106 can befurther processed by other operations including element-wise operationsto generate final output data for the convolutional operation. Theseconvolutional operations may be converted to GeMM, which may beprocessed by a systolic array or accelerator architecture, as describedin further detail below.

When executing a convolution operation, data for each receptive fieldsuch as b1 to b3 of input data 102 are fetched from on-chip or off-chipmemory to a buffer memory for computation. As known from above,receptive fields b1 to b3 partially overlap with each other. Overlappeddata among receptive fields b1 to b3 are conventionally fetched fromon-chip memory or off-chip memory and stored to the buffer memorymultiple times for a convolution operation, which causes buffer spacedeficiency or data transfer delay. Therefore, data reuse or shareschemes for overlapped data among receptive fields b1 to b3 can bebeneficial for improving overall system throughput by reducing datastored in the buffer or by minimizing data transfer bandwidth usage.Embodiments of the present disclosure can provide an acceleratorenabling efficient processing of CNN operations. Embodiments of thepresent disclosure can also provide data reuse or share scheme suitablefor executing CNN operations.

FIG. 2A illustrates an exemplary neural network acceleratorarchitecture, consistent with embodiments of the present disclosure. Inthe context of this disclosure, a neural network accelerator may also bereferred to as a machine learning accelerator or deep learningaccelerator. In some embodiments, accelerator architecture 200 may bereferred to as a neural network processing unit (NPU) architecture 200.As shown in FIG. 2A, accelerator architecture 200 can include aplurality of cores 202, a command processor 204, a direct memory access(DMA) unit 208, a Joint Test Action Group (JTAG)/Test Access End (TAP)controller 210, a peripheral interface 212, a bus 214, and the like.

It is appreciated that cores 202 can perform algorithmic operationsbased on communicated data. Cores 202 can include one or more processingelements that may include single instruction, multiple data (SIMD)architecture including one or more processing units configured toperform one or more operations (e.g., multiplication, addition,multiply-accumulate, etc.) based on commands received from commandprocessor 204. To perform the operation on the communicated datapackets, cores 202 can include one or more processing elements forprocessing information in the data packets. Each processing element maycomprise any number of processing units. According to some embodimentsof the present disclosure, accelerator architecture 200 may include aplurality of cores 202, e.g., four cores. In some embodiments, theplurality of cores 202 can be communicatively coupled with each other.For example, the plurality of cores 202 can be connected with a singledirectional ring bus, which supports efficient pipelining for largeneural network models. The architecture of cores 202 will be explainedin detail with respect to FIG. 2B.

Command processor 204 can interact with a host unit 220 and passpertinent commands and data to corresponding core 202. In someembodiments, command processor 204 can interact with host unit under thesupervision of kernel mode driver (KMD). In some embodiments, commandprocessor 204 can modify the pertinent commands to each core 202, sothat cores 202 can work in parallel as much as possible. The modifiedcommands can be stored in an instruction buffer. In some embodiments,command processor 204 can be configured to coordinate one or more cores202 for parallel execution.

DMA unit 208 can assist with transferring data between host memory 221and accelerator architecture 200. For example, DMA unit 208 can assistwith loading data or instructions from host memory 221 into local memoryof cores 202. DMA unit 208 can also assist with transferring databetween multiple accelerators. DMA unit 208 can allow off-chip devicesto access both on-chip and off-chip memory without causing a host CPUinterrupt. In addition, DMA unit 208 can assist with transferring databetween components of accelerator architecture 200. For example, DMAunit 208 can assist with transferring data between multiple cores 202 orwithin each core. Thus, DMA unit 208 can also generate memory addressesand initiate memory read or write cycles. DMA unit 208 also can containseveral hardware registers that can be written and read by the one ormore processors, including a memory address register, a byte-countregister, one or more control registers, and other types of registers.These registers can specify some combination of the source, thedestination, the direction of the transfer (reading from theinput/output (I/O) device or writing to the I/O device), the size of thetransfer unit, or the number of bytes to transfer in one burst. It isappreciated that accelerator architecture 200 can include a second DMAunit, which can be used to transfer data between other acceleratorarchitectures to allow multiple accelerator architectures to communicatedirectly without involving the host CPU.

JTAG/TAP controller 210 can specify a dedicated debug port implementinga serial communications interface (e.g., a JTAG interface) forlow-overhead access to the accelerator without requiring direct externalaccess to the system address and data buses. JTAG/TAP controller 210 canalso have on-chip test access interface (e.g., a TAP interface) thatimplements a protocol to access a set of test registers that presentchip logic levels and device capabilities of various parts.

Peripheral interface 212 (such as a PCIe interface), if present, servesas an (and typically the) inter-chip bus, providing communicationbetween the accelerator and other devices.

Bus 214 (such as a I²C bus) includes both intra-chip bus and inter-chipbuses. The intra-chip bus connects all internal components to oneanother as called for by the system architecture. While not allcomponents are connected to every other component, all components dohave some connection to other components they need to communicate with.The inter-chip bus connects the accelerator with other devices, such asthe off-chip memory or peripherals. For example, bus 214 can providehigh speed communication across cores and can also connect cores 202with other units, such as the off-chip memory or peripherals. Typically,if there is a peripheral interface 212 (e.g., the inter-chip bus), bus214 is solely concerned with intra-chip buses, though in someimplementations it could still be concerned with specialized inter-buscommunications.

Accelerator architecture 200 can also communicate with a host unit 220.Host unit 220 can be one or more processing unit (e.g., an X86 centralprocessing unit). As shown in FIG. 2A, host unit 220 may be associatedwith host memory 221. In some embodiments, host memory 221 may be anintegral memory or an external memory associated with host unit 220. Insome embodiments, host memory 221 may comprise a host disk, which is anexternal memory configured to provide additional memory for host unit220. Host memory 221 can be a double data rate synchronous dynamicrandom-access memory (e.g., DDR SDRAM) or the like. Host memory 221 canbe configured to store a large amount of data with slower access speed,compared to the on-chip memory integrated within accelerator chip,acting as a higher-level cache. The data stored in host memory 221 maybe transferred to accelerator architecture 200 to be used for executingneural network models.

In some embodiments, a host system having host unit 220 and host memory221 can comprise a compiler (not shown). The compiler is a program orcomputer software that transforms computer codes written in oneprogramming language into instructions for accelerator architecture 200to create an executable program. In machine learning applications, acompiler can perform a variety of operations, for example,pre-processing, lexical analysis, parsing, semantic analysis, conversionof input programs to an intermediate representation, initialization of aneural network, code optimization, and code generation, or combinationsthereof. For example, the compiler can compile a neural network togenerate static parameters, e.g., connections among neurons and weightsof the neurons.

In some embodiments, host system including the compiler may push one ormore commands to accelerator architecture 200. As discussed above, thesecommands can be further processed by command processor 204 ofaccelerator architecture 200, temporarily stored in an instructionbuffer of accelerator architecture 200, and distributed to correspondingone or more cores (e.g., cores 202 in FIG. 2A) or processing elements.Some of the commands may instruct a DMA unit (e.g., DMA unit 208 of FIG.2A) to load instructions and data from host memory (e.g., host memory221 of FIG. 2A) into accelerator architecture 200. The loadedinstructions may then be distributed to each core (e.g., core 202 ofFIG. 2A) assigned with the corresponding task, and the one or more coresmay process these instructions.

It is appreciated that the first few instructions received by the cores202 may instruct the cores 202 to load/store data from host memory 221into one or more local memories of the cores (e.g., local memory 2032 ofFIG. 2B). Each core 202 may then initiate the instruction pipeline,which involves fetching the instruction (e.g., via a sequencer) from theinstruction buffer, decoding the instruction (e.g., via a DMA unit 208of FIG. 2A), generating local memory addresses (e.g., corresponding toan operand), reading the source data, executing or loading/storingoperations, and then writing back results.

According to some embodiments, accelerator architecture 200 can furtherinclude a global memory (not shown) having memory blocks (e.g., 4 blocksof 8 GB second generation of high bandwidth memory (HBM2)) to serve asmain memory. In some embodiments, the global memory can storeinstructions and data from host memory 221 via DMA unit 208. Theinstructions can then be distributed to an instruction buffer of eachcore assigned with the corresponding task, and the core can processthese instructions accordingly.

In some embodiments, accelerator architecture 200 can further includememory controller (not shown) configured to manage reading and writingof data to and from a specific memory block (e.g., HBM2) within globalmemory. For example, memory controller can manage read/write data comingfrom core of another accelerator (e.g., from DMA unit 208 or a DMA unitcorresponding to another accelerator) or from core 202 (e.g., from alocal memory in core 202). It is appreciated that more than one memorycontroller can be provided in accelerator architecture 200. For example,there can be one memory controller for each memory block (e.g., HBM2)within global memory.

Memory controller can generate memory addresses and initiate memory reador write cycles. Memory controller can contain several hardwareregisters that can be written and read by the one or more processors.The registers can include a memory address register, a byte-countregister, one or more control registers, and other types of registers.These registers can specify some combination of the source, thedestination, the direction of the transfer (reading from theinput/output (I/O) device or writing to the I/O device), the size of thetransfer unit, the number of bytes to transfer in one burst, or othertypical features of memory controllers.

While accelerator architecture 200 of FIG. 2A can be used forconvolutional neural networks (CNNs) in some embodiments of the presentdisclosure, it is appreciated that accelerator architecture 200 of FIG.2A can be utilized in various neural networks, such as deep neuralnetworks (DNNs), recurrent neural networks (RNNs), or the like. Inaddition, some embodiments can be configured for various processingarchitectures, such as neural network processing units (NPUs), graphicsprocessing units (GPUs), field programmable gate arrays (FPGAs), tensorprocessing units (TPUs), application-specific integrated circuits(ASICs), any other types of heterogeneous accelerator processing units(HAPUs), or the like

FIG. 2B illustrates an exemplary core architecture, consistent withembodiments of the present disclosure. As shown in FIG. 2B, core 202 caninclude one or more operation units such as first and second operationunits 2020 and 2022, a memory engine 2024, a sequencer 2026, aninstruction buffer 2028, a constant buffer 2030, a local memory 2032, orthe like.

One or more operation units can include first operation unit 2020 andsecond operation unit 2022. First operation unit 2020 can be configuredto perform operations on received data (e.g., matrices). In someembodiments, first operation unit 2020 can include one or moreprocessing units configured to perform one or more operations (e.g.,multiplication, addition, multiply-accumulate, element-wise operation,etc.). In some embodiments, first operation unit 2020 is configured toaccelerate execution of convolution operations or matrix multiplicationoperations. An example of first operation unit 2020 will be explainedwith respect to FIG. 3 in detail.

Second operation unit 2022 can be configured to perform a poolingoperation, an interpolation operation, a region-of-interest (ROI)operation, and the like. In some embodiments, second operation unit 2022can include an interpolation unit, a pooling data path, and the like.

Memory engine 2024 can be configured to perform a data copy within acorresponding core 202 or between two cores. DMA unit 208 can assistwith copying data within a corresponding core or between two cores. Forexample, DMA unit 208 can support memory engine 2024 to perform datacopy from a local memory (e.g., local memory 2032 of FIG. 2B) into acorresponding operation unit. Memory engine 2024 can also be configuredto perform matrix transposition to make the matrix suitable to be usedin the operation unit.

Sequencer 2026 can be coupled with instruction buffer 2028 andconfigured to retrieve commands and distribute the commands tocomponents of core 202. For example, sequencer 2026 can distributeconvolution commands or multiplication commands to first operation unit2020, distribute pooling commands to second operation unit 2022, ordistribute data copy commands to memory engine 2024. Sequencer 2026 canalso be configured to monitor execution of a neural network task andparallelize sub-tasks of the neural network task to improve efficiencyof the execution. In some embodiments, first operation unit 2020, secondoperation unit 2022, and memory engine 2024 can run in parallel undercontrol of sequencer 2026 according to instructions stored ininstruction buffer 2028.

Instruction buffer 2028 can be configured to store instructionsbelonging to the corresponding core 202. In some embodiments,instruction buffer 2028 is coupled with sequencer 2026 and providesinstructions to the sequencer 2026. In some embodiments, instructionsstored in instruction buffer 2028 can be transferred or modified bycommand processor 204.

Constant buffer 2030 can be configured to store constant values. In someembodiments, constant values stored in constant buffer 2030 can be usedby operation units such as first operation unit 2020 or second operationunit 2022 for batch normalization, quantization, de-quantization, or thelike.

Local memory 2032 can provide storage space with fast read/write speed.To reduce possible interaction with a global memory, storage space oflocal memory 2032 can be implemented with large capacity. With themassive storage space, most of data access can be performed within core202 with reduced latency caused by data access. In some embodiments, tominimize data loading latency and energy consumption, SRAM (staticrandom access memory) integrated on chip can be used as local memory2032. In some embodiments, local memory 2032 can have a capacity of 192MB or above. According to some embodiments of the present disclosure,local memory 2032 be evenly distributed on chip to relieve dense wiringand heating issues.

FIG. 2C is a schematic representation of an exemplary operation unitconfiguration 2020, according to some embodiments of the presentdisclosure. Operation unit 2020 can include a first buffer 2034, asecond buffer 2036, and a processing array 2038.

First buffer 2034 can be configured to store input data. In someembodiments, data stored in first buffer 2034 can be input data (e.g.,input features) to be used in processing array 2038 for execution. Insome embodiments, the input data can be fetched from local memory 2032.First buffer 2034 can be configured to support reuse or share of data tobe used in processing array 2038. In some embodiments, input data storedin first buffer 2034 can be activation data for a convolution operation.

Second buffer 2036 can be configured to store matrix data, such as arepresentation of sparse matrix (e.g., weight matrix). For example,operation unit 2020 can read, fetch or receive the representation fromlocal memory 2032 through a memory engine (not shown, e.g., memoryengine 2024 of FIG. 2B), and store the representation in second buffer2036. In some embodiments, second buffer 2036 can be a part of orseparate from first buffer 2034. Second buffer 2036 can be any suitablememory that provides storage space for data such as matrix orrepresentation, such as a register, Dynamic Random Access Memory (DRAM),Static Random Access Memory (SRAM), or the like.

Operation unit 2020 can also include a sparse engine 2040communicatively coupled with second buffer 2036 and configured to readdata from or write data to second buffer 2036. Sparse engine 2040 mayinclude one or more decompressors, such as a first level decompressorand a second level decompressor, to decompress the representation. Insome embodiments, sparse engine 2040, which may include a first leveldecompressor or second level decompressor can be implemented as acircuitry with high processing speed. Sparse engine 2040 can read arepresentation of a sparse matrix in a neural network from second buffer2036. In some embodiments, the representation can also include one ormore sub-level BMs. Decompressors of sparse engine 2040 can includecircuitry to communicatively couple them and allow them to cooperatewith each other to decompress the representation. For example, a firstlevel decompressor and a second level decompressor can be coupled andcommunicate with each other.

Operation unit 2020 can also include processing array 2038 that can havea plurality of layers (e.g., K layers). According to some embodiments ofthe present disclosure, each layer of processing array 2038 can includea plurality of processing strings, which may perform computations inparallel. For example, first processing string included in the firstlayer of processing array 2038 can comprise a first multiplier (e.g.,dot product) 240_1 and a first accumulator (ACC) 250_1 and secondprocessing string can comprise a second multiplier 240_2 and a secondaccumulator 250_2. Similarly, i^(th) processing string in the firstlayer can comprise an i^(th) multiplier 240_i and an i^(th) accumulator250_i.

In some embodiments, processing array 2038 can perform computationsunder SIMD control. For example, when performing a convolutionoperation, each layer of processing array 2038 can execute sameinstructions with different data.

According to some embodiments of the present disclosure, processingarray 2038 shown in FIG. 2C can be included in a core (e.g., core 202 inFIG. 2A or FIG. 2B). When a number of processing strings (e.g., i numberof processing strings) included in one layer of processing array 2038 issmaller than a number of work items (e.g., B number of work items), inumber of work items can be executed by processing array 2038 andsubsequently the rest of work items (B-i number of work items) can beexecuted by the processing array 2038 in some embodiments. In some otherembodiments, i number of work items can be executed by processing array530 and the rest of work items can be executed by another processingarray 2038 in another core.

According to some embodiments of the present disclosure, processingarray 2038 can further include an element-wise operation processor (OP)260. In some embodiments, element-wise operation processor 260 can bepositioned at the end of processing strings. In some embodiments,processing strings in each layer of processing array 2038 can shareelement-wise operation processor 260. For example, i number ofprocessing strings in the first layer of processing array 2038 can shareelement-wise operation processor 260. In some embodiments, element-wiseoperation processor 260 in the first layer of processing array 2038 canperform its element-wise operation on each of output values, fromaccumulators 250_1 to 250_i, sequentially. Similarly, element-wiseoperation processor 260 in the Kth layer of processing array 2038 canperform its element-wise operation on each of output values, fromaccumulators 250_1 to 250_i, sequentially. In some embodiments,element-wise operation processor 260 can be configured to perform aplurality of element-wise operations. In some embodiments, element-wiseoperation performed by the element-wise operation processor 260 mayinclude an activation function such as ReLU function, ReLU6 function,Leaky ReLU function, Sigmoid function, Tanh function, or the like.

In some embodiments, multiplier 240 or accumulator 250 may be configuredto perform its operation on different data type from what theelement-wise operation processor 260 performs its operations on. Forexample, multiplier 240 or accumulator 250 can be configured to performits operations on integer type data such as Int 8, Int 16, and the likeand element-wise operation processor 260 can perform its operations onfloating point type data such as FP24, and the like. Therefore,according to some embodiments of the present disclosure, processingarray 2038 can further include de-quantizer 270 and quantizer 280 withelement-wise operation processor 260 positioned therebetween. In someembodiments, batch normalization operations can be merged tode-quantizer 270 because both de-quantizer 270 and batch normalizationoperations can be performed by multiplication operations and additionoperations with constants, which can be provided from constant buffer2030. In some embodiments, batch normalization operations andde-quantization operations can be merged into one operation by compiler.As shown in FIG. 2C, constant buffer 2030 can provide constants tode-quantizer 270 for de-quantization or batch normalization.

Sparse engine 2040 can provide a decompressed sparse matrix toprocessing array 2038, and processing array 2038 can perform acomputation (e.g., addition, multiplication, multiply-accumulate,convolution, or the like) on the decompressed sparse matrix. In someembodiments, processing array 2038 can read input features from fistbuffer 2034 and use them in a computation.

FIG. 2D illustrates a schematic diagram of an exemplary cloud systemincorporating accelerator architecture 200, consistent with embodimentsof the present disclosure. As shown in FIG. 2D, cloud system 230 canprovide a cloud service with artificial intelligence (AI) capabilitiesand can include a plurality of computing servers (e.g., 232 and 234). Insome embodiments, a computing server 232 can, for example, incorporate aneural network accelerator architecture 200 of FIG. 2A. Neural networkaccelerator architecture 200 is shown in FIG. 2D in a simplified mannerfor simplicity and clarity.

With the assistance of neural network accelerator architecture 200,cloud system 230 can provide the extended AI capabilities of imagerecognition, facial recognition, translations, 3D modeling, and thelike. It is appreciated that, neural network accelerator architecture200 can be deployed to computing devices in other forms. For example,neural network accelerator architecture 200 can also be integrated in acomputing device, such as a smart phone, a tablet, and a wearabledevice.

FIG. 3 illustrates an exemplary systolic array-based accelerator. Asdescribed above, such a configuration may be used to facilitateoperations of convolutional neural networks. The systolic array 300 mayinclude an N×N array of multiply-and-accumulate (MAC) units 302configured to execute a matrix multiplication. In some embodiments, MACunits 302 may be part of one or more operation units (e.g., firstoperation 2020 of FIG. 2B), which can perform multiply and accumulateoperations (e.g., via various output channels 1-k of processing array2038, shown in FIG. 2C). Activation memory 304 (e.g., first buffer 2034shown in FIG. 2C) may store an activation matrix, which may be fed intoeach row of the systolic array at a rate of one row per clock cycle. Insome embodiments, each row may receive activation data offset by oneclock cycle. Weight memory 306 (e.g., second buffer 2036, shown in FIG.2C) may store a weight matrix, which may be preloaded into the array andremains stationary throughout the computation.

Each MAC unit 302 may receive the result of a calculation performed byits neighboring units, perform an operation using the result, andcommunicate the result of the operation to its neighbors. For example, aMAC unit may receive numbers from its neighbors, perform one or morecalculations, and then pass the result its right-side or lowerneighbors. In a first clock cycle, MAC units 302 may receive anactivation stream. For example, MAC₁₁ may compute w₁₁a₁₁. In a secondclock cycle, MAC₁₂ may compute w₁₂a₁₂, which is added to the result ofw₁₁a₁₁ by an accumulator 308. In the Nth clock cycle, a MAC_(IN) mayoutput a result y₁₁=Σ_(i)w_(1i)a_(1i) where i indicates the i^(th) rowof the weight matrix and where y₁₁ is the first element in an outputmatrix stored by accumulator 308. After the matrix multiplication of theactivation matrix and weight matrix is complete, an activation functionmay be applied to the output matrices stored in the accumulators 308.

In the implementation of a systolic array as described with reference toFIG. 3, the input data (e.g., the activation matrix and weight matrix)must be staged prior to being operated on by the systolic array. Thatis, the input data must follow a pattern in order to yield an accuratecalculation. For example, systolic arrays used to do matrixmultiplication require the input to follow a particular pattern. If theinput is in the form of a matrix, each row and each column may need tobe shifted, such that, for example, the second row of the input matrixis delayed by one cycle, the third row is delayed by two cycles, and soon. This is often accomplished by using staging buffers (e.g., linebuffers) to stage both the weight and activation data, as well as theoutput data. The staging buffer may delay the input data, e.g., by aclock cycle, thereby staggering the input data. Often, the data may bestored in the staged format prior to being fed into a systolic array.The use of staging buffers is costly and inefficient, as described belowwith reference to FIG. 4.

FIG. 4 illustrates a conventional operation of a systolic array bymethod of staging input data, e.g., by using a staging buffer. In FIG.4, both weight, activation, and output memories are represented bymemory 402. Memory 402 may be accelerator memory or host memory. Insystem 400, data, e.g., activation data 404 and weight data 406, may besent from memory 402 to an input staging buffer 408. Staging buffer 408may receive activation data 404 in a series of clock cycles, denoted ast0, t1, and t2.

Each input buffer 408 may act upon received data to stage the data intoa format compatible with systolic array 410. For example, with respectto activation data 404, input buffer 408 may stagger the data, such thatthe data received by systolic array 410 is offset by one clock cycle.Thus, activation data 404 may be staggered across four clock cycles asillustrated by staged data 412. The same staging process occurs for theweight data 406. The staged weight and input data may be received bysystolic array 410, which may be configured to perform an operation,e.g., matrix multiplication, on the received data to generate outputdata, as described above with reference to FIG. 3.

An output staging buffer 414 may be configured to manipulate the dataoutput as a result of the systolic array 410 performing a computation onthe staged input data, thereby output data 416, which is data in itsoriginal, unstaged format. The formatted output data 416 may be storedin memory 402 for subsequent use in calculations.

FIGS. 5A and 5B provide another illustration of the issues associatedwith storing data for input into a systolic array. In FIG. 5A, theexemplary input data 502 is stored in a normal format. Data 502 may thenbe staged using a buffer 504 and input to systolic array 506. Systolicarray 506 may output data, which is then operated on by an outputstaging buffer, e.g., output staging buffer 412, to un-stage the outputmatrix. Thus, the data may be saved in its original format, for example,with each address storing one column. Although there is a gap in thetime domain between the two output matrices, they can be stored insequence in memory and be accessed in sequence as an input matrix insubsequent operations. In some instances, the output matrix columns maynot be stored in memory sequentially, but they can be accessedsequentially one column per cycle by nonconsecutive addresses.

FIG. 5B illustrates an implementation in which the input data 502 isstored in memory in a pre-staged format, which is directly inputted intosystolic array 506. In this situation, staging buffers may be removed.The time domain gap in the output of systolic array 506 may be shifted,due to the pre-staged data, such that the gap cannot be removed. This isbecause each column contains valid values as well as placeholder values.Thus, the placeholder values must be stored in memory, which results instorage overhead. In addition, this method requires an additionalprocess to remove the placeholder values prior to additional operations.

In the examples illustrated by both FIGS. 5A and 5B, problems arise fromthe fact that the output is not stored in the same format as the input.Thus, systems may not perform N×M * M×N GeMM operations consecutively.These systems are limited to N×N * N×N GeMM operations, resulting in alow utilization rate. Further, because gaps in data are stored inmemory, these methods of storing output data are inefficient, storingunnecessary data.

FIG. 6 illustrates the deficiencies associated with removing the stagingcomponents from the system entirely, for example, by storing the inputdata in a pre-staged format. For example, process 602 illustrates aninput string received from memory, e.g., memory 402. In the absence ofstaging buffers, the input data must be stored in a pre-staged format.Input string 604 may be comprised of two blocks, e.g., input matrixblock 1 and input matrix block 2. Each segment 606 may represent aportion of input data that may correspond to an address in memorystoring a portion of input string 604. For example, each segment maystore one byte. In other embodiments, each segment may contain more thanone byte of data, or any other amount of data. Input string 604 may bestored such that the data of each input block is shifted in order toreach the systolic array, e.g., array 410, at the appropriate clockcycle. Placeholder values are denoted here as “x.”

Process 608 illustrates the output in memory of output matrix block 1and output matrix block 2. Notably, the output is not stored in the sameformat as the input. The output may include several placeholder values,thus data efficiency using this method is low. In some examples, dataefficiency may be lower in output memory than in input memory as the gapin the original data cannot be removed. This gap may be introduced as aresult of the removal of the output staging buffer. For example, ifoutput matrix 1 (N*N) is the output from a N*M * M*N GeMM, and outputmatrix 2 (N*N) is the output from another N*M * M*N GeMM; due to theremoval of output staging buffer, the two output matrices will haveplaceholder values. In order to use the two output matrices as input foranother N*2N, 2N *N GeMM operation, the placeholders will requirespecial handling and additional processing. Accordingly, this method ofstoring data is associated with additional overhead costs due to thestorage of the extraneous placeholder data.

The embodiments of the present disclosure address problems associatedwith both staging approaches for systolic arrays. For example, theembodiments of the present disclosure use masked writes to combineoutput matrices such that the output matrices may be merged and storedin memory. The merged output may then be used directly as the input forfurther matrix multiplication by the systolic array. Saving the maskedoutput data enables this data to be merged and used for subsequentcomputations, thereby reducing the overhead and increasing theefficiency associated with operating a systolic array-based accelerator.

The embodiments of the present disclosure describe a systolic arrayfriendly data placement and control method in which data are not storedin original format, but in a staged format. In some embodiments,systolic array output is written back to memory with data masks to mergewith previous N×N data blocks. For example, a data mask module mayreceive a pattern generation signal from a controller and automaticallygenerate mask bits for 2n-1 consecutive writes. The data mask generatormay generate a data mask pattern according to the input from thecontroller to support different data formats, e.g., (int8, fp16, bf16,fp32, etc).

FIG. 7 illustrates a block diagram of a system 700 for staging inputdata for a systolic array. The system 700 may include a compute engine702 containing a systolic array 704, a controller 706, a data maskgenerator 708, and a memory sub-system 710. System 700 eliminates theoverheard associated with traditional methods for storing staged inputand output data and enables subsequent computations to be completedusing output data, thereby increasing the utilization rate over thepreviously described systems.

Compute engine 702 may include systolic array 704 or other components,such as an accumulator. Systolic array 704 may be similar to systolicarray 300, described with reference to FIG. 3. In the embodiments of thepresent disclosure, staging buffers may not be required to reshapeinput/output data of the compute engine 702.

Controller 706 may coordinate compute/memory and data mask module 708during write operations. In the embodiments of the present disclosure,the controller 706 is a dedicated module. In other embodiments,controller 706 may be an external processor, for example, a CPU. Controlsequences may be instructions, which are compiled before the run ordetermined at runtime based on the current GeMM operation. Accordingly,disclosed embodiments may be modified to accommodate a number of dataformats.

Controller 706 may also communicate with Memory sub-system 710.Controller 706 may receive GeMM instructions from another externalmodule. These instruction can contain, for example, GeMM operationoperand addresses, matrix sizes, and data type information. Thecontroller 706 may orchestrate the systolic array computation and memoryread/write. Controller 706 may also send read commands and addresses tomemory sub-system 710. In response, memory sub-system 710 may outputdata to systolic array 704 as input. The controller 706 may sendaddresses and read commands to memory sub-system 710 every cycle toproduce the systolic array input columns. During output cycles,controller 706 may issue write commands and addresses to memorysub-system 710 to write systolic array output columns to specifiedmemory address locations. Additionally, controller 706 may send computecontrol information to compute engine 702 to cause compute engine 702 tostart to compute, accumulate, and output matrices in specific cycles.

Memory sub-system 710 may be a single memory for both weight andactivation. Memory sub-system 710 can be separate weight and activationmemories. Memory sub-system 710 may include separate weight, activation,and output memories. In some embodiments, memory sub-system 710 maysupport masked writes. For example, memory sub-system 710 may be a DRAM.Proprietary memory may also be designed to support masked writes.

Moreover, memory sub-system 710 may include an input/output (IO)component to receive and send bytes to compute engine 702. This IOcomponent, or data interface, may be the same width as the systolicarray output or the byte-wise mask generated by data mask generator 708.

If memory sub-system 710 has more than one physical memory, eachphysical memory can either have its own data mask generator, or eachphysical memory may share one data mask generator. The memory that isconnected with data mask generator may supportsystolic-array-output-merge-writes.

Multiple data mask generators may be required to merge output, e.g., towrite output into memory efficiently. If a memory is dedicated to weightin all scenarios, then a data mask generator may not be required forthis memory because there is no need to merge weight data. However, insome embodiments, if a memory is only allocated for weight in aparticular operation/run, and may be allocated otherwise, e.g., asactivation memory in another operation/run, then data masks may berequired to be generated for this memory.

In the embodiments of the present disclosure, a data mask generator maybe shared by multiple memories as long as the memories do not need toperform masked writes simultaneously. This rationale may also apply tosystems having separate weight, activation, and output memories.

In the embodiments of the present disclosure, memory sub-system 710 maycorrespond to local memory 2032 shown in FIG. 2B. Compute engine 702 maybe the same as or similar to first operation unit 2020, where controller706 or data mask generator 708 may be incorporated into one or morecomponents of core 202 (e.g., sequencer 2026, instruction buffer 2028,second operation unit 2022, etc.) or can be separate components of core202.

FIG. 8 illustrates two strides of data mask generator, e.g., data maskgenerator 708. As used herein, two strides means the generated maskpattern shifts two bits for every output. The two strides may be used,for example, when each number in the matrix requires two bytes torepresent (for example integer 16, or floating point 16 data type). Thetwo stride may enable the data mask generator to support multiple datatypes. In FIG. 8, an exemplary N bit data mask output is shown, whereone bit represents a mask for one byte of data. The width of data maskoutput may match the output of compute 702 (e.g., the output of systolicarray 704), output width, or memory IO width.

In the embodiments of the present disclosure, data mask generator 708may use shift registers. For example, the shift register input may be“1” for m cycles and “0” for m-1 cycles, where m is the output matrixsize. The shift stride “s” may equal the number of bytes of output data.For example, if the output type is integer 8, then stride is 1. Ifoutput type is integer 16, or float-point 16, or bfloat 16, the strideis 2. If output type is fp32/int32, the stride is 4, and so on, e.g., Nequals m*s. In other embodiments, data mask generator 708 may behardware or other circuitry configured to generate a bit mask. Forexample, data mask generator may be circuitry configured to receive apattern generation signal from the controller (e.g., controller 706),and, based on the received signal, generate a mask corresponding to eachsegment of a plurality of segments of the staged output data.

FIG. 9A is another exemplary systolic array-based accelerator 900.Accelerator 900 may include a memory 902, which supports read writeoperations. In the embodiments of the present disclosure, the memory maysupport masked write operations, meaning the memory may accept maskedsequential bits. As discussed with reference to FIG. 7, a data maskgenerator, e.g., data mask generator 904, may receive input from acontroller. The input may be a pattern generation signal configured tocause data mask generator 904 to produce a bit-wise mask.

A multiplexer (MUX) 906 may receive segments of a matrix block of datafrom a systolic array (not shown) and mask data from data mask generator904. For a byte where the masked bit is 1, the corresponding data of thesegment is read through to memory 902. If the masked bit of thecorresponding byte is 0, the corresponding data of the segment isignored and will not be written to the memory.

This enables masked segment data from MUX 906 to be merged and stored,such as merging tail data associated with a matrix block with head dataassociated with another matrix block. In particular, when memory 902initially receives masked segment data “x 8 6” of tail 908 of firstoutput data 910, this masked segment data can be merged with a head ofdifferent output data. In a subsequent operation, if “a x x” is receivedat memory 902 (i.e., data associated with mask “100”), it is merged withdata at an address associated with a mask “011.” Thus, a tail 908 of thefirst output 910 may be merged with a head 912 of the second output 914.Using this method, first output 910 and second output 914 may be mergedand stored as merged data 916.

Components of accelerator 900 may be integrated into NPU architecture200, for example, as part of a core 202, shown in FIG. 2B. For example,memory 902 may correspond to local memory 2032, while data maskgenerator 904 and mux 906 may be additional components of core 202 or offirst operational unit 2020.

FIG. 9B illustrates an exemplary data structure as shown in FIG. 9A withreference to memory 902. For example, memory 902 may receive addressesand data output by MUX 906, which includes output data segments based ontheir associated mask. In some embodiments, output by MUX 906 caninclude the associated masks. In some embodiments, the associated masksare not included in the output as memory 902 may have been previouslyconfigured to associate an incoming segment with a data mask. Forexample, if each matrix block includes five segments, then the firstincoming segment of each block would correspond with mask “1 0 0,” thesecond segment with “1 1 0,” the third segment with “1 1 1,” the fourthsegment with “0 1 1,” and the fifth segment “0 0 1.” In the exampleshown in FIG. 9B, first output data 910 may include three data segments,with one segment associated with addr1 and mask “1 1 1,” one segmentassociated with addr2 and mask “0 1 1,” and one segment associated withmask “0 0 1.”

Memory 902 may subsequently receive a second output 914 and may aligndata segments based on complementary masks. For example, the segment ataddr2 having mask “0 0 1” may be matched with the complementary segmentof the second output, which is associated with a mask of “1 1 0.” Thecomplementary segments are then merged such that each address storesuseful information, e.g., no placeholder values. Thus, the resultingcompressed output includes data from both the first and second outputdata 910 and 914, respectively. Subsequently received data may then bemerged with the tail 920 of the resulting output data stream 916.

It is appreciated that while FIGS. 9A-9B show segments having threebytes of data and three corresponding mask bits, any number of bytes persegment and any number of corresponding mask bits can be used. Moreover,while the exemplary embodiments above describe a mask bit correspondingto a byte of data in a segment, it is appreciated that the mask bit cancorrespond to any number of bits of data in a segment.

FIG. 10 illustrates the improved efficiency of disclosed embodiments forsystolic array-friendly data placement. Given an exemplary output 1002,using an accelerator such as that described with reference to FIG. 3,the memory must perform a number of reads and writes. For example, usingtraditional methods for staging input and output data, a system mayrequire additional components, e.g., a component with temporary memory,to merge the data in memory with the mask bits. The system may read thecolumn with placeholder to this temporary memory, merge the column withother data, and then write the merged data to DRAM. Thus, this merge isperformed but requires additional read operations and components.

In contrast, using the methodology described with reference to FIGS. 7and 9, the memory is only required to perform write operations, therebyimproving efficiency. For example, the merging of data occurs in memory,e.g., memory sub-system 710 or memory 902, with the control of carefullygenerated addresses and masking write mechanism. No additionalcomponents or read operations are required to execute the merge. Intraditional systems, the output columns will be written to addr1, addr2,addr3, and the next matrix output will continue to be written to addr4,addr5, addr6. Without the masking bits, the system cannot write to apreviously used address because it will overwrite useful information.However, disclosed embodiments allow the system to write to previouslyused addr2, addr3 by only overwriting the placeholder values stored atthese addresses. Accordingly, disclosed embodiments enable output datato be merged without losing useful information.

FIG. 11 is a flowchart of an exemplary method 1100 for generating datahaving systolic array-friendly placement, according to some embodimentsof the present disclosure. Method 1100 can be implemented by neuralnetwork architecture 200 of FIG. 2A or FIG. 2C or core 202 of FIG. 2C.Moreover, method 1100 can also be implemented by a computer programproduct, embodied in a computer-readable medium, includingcomputer-executable instructions, such as program code, executed bycomputers.

As shown in FIG. 11, at step 1110, a systolic array (e.g., processingarray 2038, shown in FIG. 2C) may receive staged input data. Aspreviously described, the input data may be weight data or activationdata. The input data may be generated by shifting an input data streamsuch that each column of input data is delayed by a clock cycle.

At step 1120, the systolic array (e.g., processing array 2038, shown inFIG. 2C) may perform an operation using the staged input data, therebygenerating a first output matrix and a second output matrix. Theoperation may be, for example, a GeMM operation.

At step 1130, the system may generate a pattern generation signal. Thepattern generation signal may be generated by a controller (e.g.,controller 706, shown in FIG. 7), or by other circuitry configured togenerate a signal during each clock cycle.

At step 1140, the system may generate a mask based on the patterngeneration signal. The mask may be generated by circuitry, for example,by a shift register, or by a data mask generator (e.g., data maskgenerator 708, shown in FIG. 7). The pattern may be based on the widthof the systolic array output and may be incremented based on the patterngeneration signal.

At step 1150, the system may store merged output data including thefirst output matrix and the second output matrix. The output data may begenerated by merging multiple data segments based on the mask associatedwith each data segment. For example, as described with reference to FIG.9B, the tail of a first output stream may include segments storingplaceholder values. This tail may be merged with the head of asubsequent output stream, where one or more segments store placeholdervalues in complementary positions to those in the tail of the firstoutput stream. Accordingly, the head and tail may be merged such togenerate output data that does not include any placeholder values. Themerged data may be stored in a memory (e.g., memory sub-system 710,shown in FIG. 7) of the system or may be operated on by the systolicarray to perform subsequent operations.

In the embodiments of the present disclosure, a method of implementing asystolic array, e.g., processing array 2038, may include fetching one ormore batches of data in a first memory (e.g., memory 2032 shown in FIGS.2B and 2C). The method may include regrouping the one or more batches ofdata into multiple work items. For example, a first work item maypartially overlap one or more work items among the multiple work items.For example, the data stored in the one or more segments of the inputdata stream may partially or fully overlap (e.g., merged). The workitems may be broadcasted to a processing array (e.g., processing array2038 shown in FIG. 2C) having a number of processing strings. The firstwork item may be transferred to two or more processing strings of theprocessing array.

The embodiments of the present disclosure may include deleting a portionof the one or more batches of data when the portion of the one or morebatches of data is determined not to be used in a predetermined timeperiod. For example, the data may be deleted from the memory (e.g.,memory 2032 of FIG. 2C). The system may also generate a number ofoutputs by the number of processing strings in parallel.

Embodiments herein include database systems, methods, and tangiblenon-transitory computer-readable media. The methods may be executed, forexample, by at least one processor that receives instructions from atangible non-transitory computer-readable storage medium (such as of ahost system having host unit 220 and host memory 221 of FIG. 2A).Similarly, systems consistent with the present disclosure may include atleast one processor and memory, and the memory may be a tangiblenon-transitory computer-readable storage medium. As used herein, atangible non-transitory computer-readable storage medium refers to anytype of physical memory on which information or data readable by atleast one processor may be stored. Examples include random access memory(RAM), read-only memory (ROM), volatile memory, non-volatile memory,hard drives, CD ROMs, DVDs, flash drives, disks, registers, caches, andany other known physical storage medium. Singular terms, such as“memory” and “computer-readable storage medium,” may additionally referto multiple structures, such a plurality of memories orcomputer-readable storage media. As referred to herein, a “memory” maycomprise any type of computer-readable storage medium unless otherwisespecified. A computer-readable storage medium may store instructions forexecution by at least one processor, including instructions for causingthe processor to perform steps or stages consistent with embodimentsherein. Additionally, one or more computer-readable storage media may beutilized in implementing a computer-implemented method. The term“non-transitory computer-readable storage medium” should be understoodto include tangible items and exclude carrier waves and transientsignals.

As used herein, unless specifically stated otherwise, the term “or”encompasses all possible combinations, except where infeasible. Forexample, if it is stated that a database may include A or B, then,unless specifically stated otherwise or infeasible, the database mayinclude A, or B, or A and B. As a second example, if it is stated that adatabase may include A, B, or C, then, unless specifically statedotherwise or infeasible, the database may include A, or B, or C, or Aand B, or A and C, or B and C, or A and B and C.

The embodiments may further be described using the following clauses:

-   -   1. An accelerator comprising:    -   a systolic array comprising a plurality of operation units,        wherein the systolic array is configured to receive staged input        data and perform operations using the staged input to generate        staged output data, the staged output data comprising a number        of segments;    -   a controller configured to execute one or more instructions to        generate a pattern generation signal;    -   a data mask generator including circuitry configured to:    -   receive the pattern generation signal from the controller, and    -   based on the received signal, generate a mask corresponding to        each segment of the staged output data; and    -   a memory configured to store the staged output data using the        generated masks.    -   2. The accelerator of clause 1, wherein the data mask generator        comprises a shift register.    -   3. The accelerator of clause 1 or 2, wherein the pattern        generation signal causes the data mask generator to execute a        stride based on a type of the output data.    -   4. The accelerator of any of clauses 1-3, wherein the mask has a        width equal to an output width of the systolic array.    -   5. The accelerator of any of clauses 1-4, wherein the memory is        a dynamic random-access memory.    -   6. The accelerator of any of clauses 1-5, wherein each of the        plurality of processing units is a multiply-accumulate unit.    -   7. The accelerator of any of clauses 1-6, wherein the data mask        generator is further configured to generate the mask for 2n-1        consecutive writes.    -   8. The accelerator of any of clauses 1-7, wherein the memory is        further configured to store merged output data, wherein the        output data is merged with previous data output by the systolic        array based on the mask associated with each segment of the        staged output data.    -   9. The accelerator of clause 8, wherein the systolic array is        further configured to: receive the merged output data, and    -   perform a subsequent calculation using the merged output data.    -   10. The accelerator of clause 8, wherein the staged output data        comprises a first data stream having a tail segment associated        with a first mask and a second data stream having a head segment        associated with a second mask.    -   11. The accelerator of clause 10, wherein the merged output data        is generated by combining the tail segment of the first data        stream and the head segment of the second data stream based on        the combination of the first and second masks.    -   12. A method for systolic array-friendly data placement,        comprising:    -   receiving, by a systolic array, staged input data,    -   performing an operation using the staged input data, thereby        generating a first output matrix and a second output matrix;    -   generating a pattern generation signal;    -   generating a mask based on the pattern generation signal; and    -   storing merged output data comprising the first output matrix        and the second output matrix.    -   13. The method of clause 12, wherein the operation performed by        the systolic array comprises a general matrix multiply (GeMM)        operation.    -   14. The method of either of clause 12 or 13, wherein the staged        input data comprises one of: activation data or weight data.    -   15. The method of any of clauses 12-14, further comprising:    -   performing, by the systolic array, a second operation on the        merged output data.    -   16. The method of any of clauses 12-15, the staged input data is        generated by shifting input data such that each column of input        data is delayed by a clock cycle.    -   17. The method of any of clauses 12-16, wherein the pattern        generation signal is based on a type of data output by the        systolic array.    -   18. The method of any of clauses 12-17, further comprising:    -   deleting a portion of the one or more batches of data when the        portion of the one or more batches of data is determined not to        be used in a predetermined time period    -   19. The method of any of clauses 12-18, further comprising:        generating a plurality of outputs by the plurality of processing        strings in parallel.    -   20. A non-transitory computer readable storage medium storing a        set of instructions that are executable by at least one        processor of a computing device to cause the computing device to        perform a method for executing a convolutional neural network        operation, the method comprising:    -   fetching one or more batches of data in a first memory;    -   regrouping the one or more batches of data into multiple work        items, wherein a first work item partially overlaps one or more        work items among the multiple work items; and    -   broadcasting the multiple work items to a processing array        comprising a plurality of processing strings, wherein the first        work item is transferred to two or more processing strings of        the processing array.

In the foregoing specification, embodiments have been described withreference to numerous specific details that can vary from implementationto implementation. Certain adaptations and modifications of thedescribed embodiments can be made. Other embodiments can be apparent tothose skilled in the art from consideration of the specification andpractice of the invention disclosed herein. It is intended that thespecification and examples be considered as exemplary only, with a truescope and spirit of the invention being indicated by the followingclaims. It is also intended that the sequence of steps shown in figuresare only for illustrative purposes and are not intended to be limited toany particular sequence of steps. As such, those skilled in the art canappreciate that these steps can be performed in a different order whileimplementing the same method.

What is claimed is:
 1. An accelerator comprising: a systolic arraycomprising a plurality of operation units, wherein the systolic array isconfigured to receive staged input data and perform operations using thestaged input to generate staged output data, the staged output datacomprising a number of segments; a controller configured to execute oneor more instructions to generate a pattern generation signal; a datamask generator including circuitry configured to: receive the patterngeneration signal from the controller, and based on the received signal,generate a mask corresponding to each segment of the staged output data;and a memory configured to store the staged output data using thegenerated masks.
 2. The accelerator of claim 1, wherein the data maskgenerator comprises a shift register.
 3. The accelerator of claim 1,wherein the pattern generation signal causes the data mask generator toexecute a stride based on a type of the output data.
 4. The acceleratorof claim 1, wherein the mask has a width equal to an output width of thesystolic array.
 5. The accelerator of claim 1, wherein the memory is adynamic random-access memory.
 6. The accelerator of claim 1, whereineach of the plurality of processing units is a multiply-accumulate unit.7. The accelerator of claim 1, wherein the data mask generator isfurther configured to generate the mask for 2n-1 consecutive writes. 8.The accelerator of claim 1, wherein the memory is further configured tostore merged output data, wherein the output data is merged withprevious data output by the systolic array based on the mask associatedwith each segment of the staged output data.
 9. The accelerator of claim8, wherein the systolic array is further configured to: receive themerged output data, and perform a subsequent calculation using themerged output data.
 10. The accelerator of claim 8, wherein the stagedoutput data comprises a first data stream having a tail segmentassociated with a first mask and a second data stream having a headsegment associated with a second mask.
 11. The accelerator of claim 10,wherein the merged output data is generated by combining the tailsegment of the first data stream and the head segment of the second datastream based on the combination of the first and second masks.
 12. Amethod for systolic array-friendly data placement, comprising:receiving, by a systolic array, staged input data, performing anoperation using the staged input data, thereby generating a first outputmatrix and a second output matrix; generating a pattern generationsignal; generating a mask based on the pattern generation signal; andstoring merged output data comprising the first output matrix and thesecond output matrix.
 13. The method of claim 12, wherein the operationperformed by the systolic array comprises a general matrix multiply(GeMM) operation.
 14. The method of claim 12, wherein the staged inputdata comprises one of: activation data or weight data.
 15. The method ofclaim 12, further comprising: performing, by the systolic array, asecond operation on the merged output data.
 16. The method of claim 12,wherein the staged input data is generated by shifting input data suchthat each column of input data is delayed by a clock cycle.
 17. Themethod of claim 12, wherein the pattern generation signal is based on atype of data output by the systolic array.
 18. The method of claim 12,further comprising: deleting a portion of the one or more batches ofdata when the portion of the one or more batches of data is determinednot to be used in a predetermined time period.
 19. The method of claim12, further comprising: generating a plurality of outputs by theplurality of processing strings in parallel.
 20. A non-transitorycomputer readable storage medium storing a set of instructions that areexecutable by at least one processor of a computing device to cause thecomputing device to perform a method for executing a convolutionalneural network operation, the method comprising: fetching one or morebatches of data in a first memory; regrouping the one or more batches ofdata into multiple work items, wherein a first work item partiallyoverlaps one or more work items among the multiple work items; andbroadcasting the multiple work items to a processing array comprising aplurality of processing strings, wherein the first work item istransferred to two or more processing strings of the processing array.